We are searching for a curious and passionate team player who is a quick learner and has the ability to pivot as needed. The ideal candidate should be a clear and personable communicator who can work closely with our business and architects. They should have experience developing proof of concepts and MVP products within a large organization and thrive in a fast-paced, agile environment. Additionally, the candidate should be capable of bringing modern solutions to the team/project and identifying necessary changes.
Key Experience Required:
- Experience with no-sql databases
- Experience with data migration projects (moving from on-prem to cloud)
Must-Haves:
- 3+ years of recent experience as a Data Engineer in an Azure environment (not heavy AWS)
- Proficient in scripting with Python, as well as other languages such as PySpark, Scala, and SQL
- Strong experience building pipelines with Azure Databricks and other Azure services, such as Azure Data Factory and Azure SQL Database
- Experience with data integration, transformation, and orchestration frameworks such as Airflow, dbt, Fivetran, or similar
- Clear, personable communicator to work closely with the business
- Experience in a large organization and ability to work in a fast-paced, agile environment
Job Description:
- Design, develop, and implement end-to-end data solutions using Azure Databricks and other relevant technologies
- Build and maintain scalable data pipelines to extract, transform, and load large volumes of data from various sources
- Collaborate with cross-functional teams to gather and analyze requirements, then translate business needs into optimized data models
- Ensure data quality and integrity by implementing data validation and cleansing processes
- Optimize data processing and storage for improved performance and efficiency
- Conduct data profiling, analysis, and modeling to identify opportunities for data-driven insights and recommendations
- Monitor and troubleshoot data pipelines and processes to ensure continuous data flow and availability
Please let me know if you have any questions or need further information to assist in the candidate search. I am here to help and would be happy to provide any additional details.
